Intelligent campus management system based on artificial intelligence algorithm

Abstract

The embodiment of the invention discloses an intelligent campus management system based on an artificial intelligence algorithm, and relates to the technical field of artificial intelligence. The system comprises an acquisition module used for acquiring a monitoring video acquired for a target area and a monitoring instruction for monitoring a target event type; the video frame extraction module is used for carrying out key frame extraction on the monitoring video to obtain one or more target video frames; the video content analysis module is used for inputting a target video frame and the monitoring instruction into a trained video content analysis model for video content feature extraction and event classification to obtain a video content analysis text and an event classification result corresponding to the target video frame; wherein the monitoring instruction is used for indicating the video content analysis model to carry out a key attention direction of video content feature extraction and event classification, and the video content analysis model comprises a video frame feature extraction network, an event feature mapping network, a content analysis network and an event classification network.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a smart campus management system based on artificial intelligence algorithms. Background Art

[0002] The smart campus management system is an intelligent monitoring system based on image data processing. The system can analyze the surveillance video images on campus in real time, automatically identify and record various activities on campus, and thus improve the level of campus security management. For example, the system can monitor abnormal behaviors on campus (such as fighting, theft, safety accidents, etc.), issue alarms in time and notify relevant personnel to handle them. In addition, the system can also be used for student attendance statistics, teacher teaching evaluation, campus traffic flow analysis and other aspects, providing strong technical support for the construction of smart campuses.

[0003] However, since there are certain differences in the identification methods corresponding to different events, it is usually necessary to use multiple algorithms and processing logics to realize the identification of each event separately. This will not only increase the complexity of the system, but also increase the development cost of the system (for example, it is necessary to build and train different models separately).

[0004] Based on this, it is necessary to study a more intelligent smart campus management system to simplify the system structure, improve the overall performance of the system, and reduce the development cost of the system. Summary of the invention

[0005] To solve the above problems, the present invention provides a smart campus management system based on artificial intelligence algorithm, which includes:

[0006] An acquisition module is used to acquire the surveillance video collected for the target area and the monitoring instructions for monitoring the target event type;

[0007] A video frame extraction module is used to extract key frames from the surveillance video to obtain one or more target video frames;

[0008] A video content analysis module, used to input the one or more target video frames and the monitoring instructions into a trained video content analysis model to extract video content features and classify events, and obtain video content analysis texts and event classification results corresponding to the one or more target video frames;

[0009] Among them, the monitoring instruction is used to indicate the key focus areas of the video content parsing model for video content feature extraction and event classification. The video content parsing model includes a video frame feature extraction network, an event feature mapping network, a content parsing network and an event classification network. The video content parsing model is obtained by performing a first supervised training of aligning video frame features and frame feature description text on the video frame feature extraction network, event feature mapping network and content parsing network of the initial video content parsing model through a graphic sample training set, and performing a second supervised training of video content feature extraction and event classification on the initial video content parsing model through a monitoring instruction sample training set.

[0010] In some embodiments, the video frame extraction module is specifically used to: based on the monitoring instruction, take the video frame whose frame difference is greater than the corresponding preset frame difference threshold as a key frame.

[0011] In some embodiments, the video frame feature extraction network is used to extract features from the one or more target video frames to obtain video frame features corresponding to each target video frame; the event feature mapping network is used to convert the video frame features into frame feature description text in a text embedding space; the content parsing network is constructed based on a pre-trained generative language model to obtain video content parsing text and event content features corresponding to each target video frame based on the frame feature description text and the monitoring instructions; the event classification network is used to perform classification and identification based on the event content features to obtain the event classification result.

[0012] In some embodiments, the event feature mapping network includes a first mapping layer and a second mapping layer, and the event feature mapping network is specifically used for:

[0013] Inputting the video frame features into the first mapping layer for slice-level feature mapping, so as to map the video frame features to a word embedding space in units of slices to obtain slice embedding features;

[0014] The slice embedding feature is input into the second mapping layer to mine and extract the previous and next associated information, so as to obtain the frame feature description text.

[0015] In some embodiments, the video frame feature extraction network is obtained by performing image-text matching training on a pre-configured indication information feature extraction unit and a video frame feature extraction unit based on a self-attention mechanism through an image-text sample training set.

[0016] In some embodiments, the system further includes a training module, wherein the training module is specifically configured to:

[0017] Obtaining a first image and text sample training set corresponding to a conventional event, a second image and text sample training set corresponding to a concerned event, and an instruction information training set consisting of a plurality of monitoring instructions;

[0018] Based on the first image-text sample training set corresponding to the regular event and the second image-text sample training set corresponding to the event of interest, the video frame feature extraction network, the event feature mapping network and the content analysis network of the initial video content analysis model are subjected to first supervised training for aligning video frame features and frame feature description texts, and in the first supervised training process, the network parameters of the content analysis network are fixed and the network parameters of the video frame feature extraction network and the event feature mapping network are adjusted until a first preset training condition is met;

[0019] Based on the second image and text sample training set corresponding to the event of interest and the monitoring instruction sample training set formed by the indication information training set, a second supervised training of video content feature extraction and event classification is performed on the initial video content parsing model that meets the first preset training condition, and in the second supervised training process, the network parameters of the video frame feature extraction network and the event feature mapping network are fixed and the network parameters of the content parsing network and the event classification network are adjusted until the second preset training condition is met;

[0020] The initial video content parsing model that meets the second preset training condition is used as the trained video content parsing model.

[0021] In some embodiments, the first graphic sample training set corresponding to the conventional event includes a first sample video frame, a first focus direction information, and a first content label text corresponding to the first sample video frame, and the second graphic sample training set corresponding to the focus event includes a second sample video frame, a second focus direction information, and a second content label text corresponding to the second sample video frame, the first focus direction information and the second focus direction information are both used to guide the content parsing network to focus on the direction when performing frame content parsing, the first content label text is the expected parsing result corresponding to the video frame content description based on the first focus direction information, and the second content label text is the expected parsing result corresponding to the video frame content description based on the second focus direction information; the first supervised training process includes the following steps:

[0022] Using the first sample video frame and / or the second sample video frame as input of the video frame feature extraction network to perform video frame feature extraction, to obtain first sample video frame features corresponding to the first sample video frame and / or second sample video frame features corresponding to the second sample video frame;

[0023] Using the first sample video frame feature and / or the second sample video frame feature as input of the event feature mapping network to perform event feature mapping, thereby mapping the first sample video frame feature and / or the second sample video frame feature to a text embedding space corresponding to the content parsing network, and obtaining a first frame feature description text corresponding to the first sample video frame and / or a second frame feature description text corresponding to the second sample video frame;

[0024] Using the data pair consisting of the first frame feature description text and the first focus direction information and / or the data pair consisting of the second frame feature description text and the second focus direction information as inputs of the content parsing network to perform frame content parsing, and obtaining a first sample parsing result corresponding to the first sample video frame and / or a second sample parsing result corresponding to the second sample video frame;

[0025] Determine a first network loss value according to a difference between the first sample parsing result and the first content label text and a difference between the second sample parsing result and the second content label text;

[0026] The network parameters of the content parsing network are fixed, and the network parameters of the video frame feature extraction network and the event feature mapping network are optimized according to the first network loss value, until the first preset training condition is met and the first supervised training is completed.

[0027] In some embodiments, the second graphic sample training set corresponding to the event of interest also includes an event classification label corresponding to the second sample video frame, the instruction information training set includes sample monitoring instructions and output result annotations corresponding to the sample monitoring instructions, and the second supervised training process includes the following steps:

[0028] Using the second sample video frame as an input of a video frame feature extraction network that meets the first preset training condition to extract video frame features, thereby obtaining third sample video frame features;

[0029] Using the third sample video frame feature as an input of an event feature mapping network that meets the first preset training condition to perform event feature mapping, and obtaining a third frame feature description text;

[0030] Using the data pair consisting of the second frame feature description text and the second focus direction information, and the sample monitoring instruction as inputs of the content parsing network to perform frame content parsing, and obtain a third event content feature, a third sample parsing result, and an output result text;

[0031] Inputting the third event content feature into the event classification network for classification and identification to obtain a sample event classification result;

[0032] Determine a second network loss value according to the third sample parsing result, the second content label text, the output result text, the output result annotation, the sample event classification result, and the event classification label;

[0033] The network parameters of the video frame feature extraction network and the event feature mapping network are fixed, and the network parameters of the content parsing network and the event classification network are optimized according to the second network loss value, until the second preset training condition is met, and the second supervised training is completed.

[0034] In some embodiments, the first focus direction information and the second focus direction information both include monitoring instructions for at least one direction, wherein the monitoring instructions for each direction include indication information of multiple dimensions, and the indication information of multiple dimensions include intra-frame target recognition indication, frame picture content attribute extraction indication and frame picture content derivation indication.

[0035] In some embodiments, the intra-frame object recognition indication is used to guide the content parsing network to perform intra-frame object recognition;

[0036] The frame picture content attribute extraction instruction is used to guide the content parsing network to extract and describe the frame picture content attribute, and the frame picture content attribute includes at least one of a time attribute, a scene attribute, and an activity attribute;

[0037] The frame picture content derivation indication is used to guide the content parsing network to perform frame picture content derivation and description.

[0038] The beneficial effects that may be brought about by the smart campus management system based on artificial intelligence algorithms provided in the embodiments of this specification include at least:

[0039] (1) By analyzing the content and classifying events of the surveillance videos collected in the target area, the intelligent and automated level of the smart campus management system can be realized, and the timeliness and comprehensiveness of campus safety management can be improved;

[0040] (2) A video content parsing model is constructed by means of a video frame feature extraction network, an event feature mapping network, a content parsing network and an event classification network. By performing the first supervised training and the second supervised training on the model, the trained video content parsing model can recognize different target events under the guidance of monitoring instructions, avoiding the need to model and train different event types separately, thereby reducing the complexity of the system and the development and maintenance costs to a certain extent and improving the intelligence of the system.

[0041] (3) By guiding the video content parsing model through monitoring instructions to perform feature analysis based on this in the subsequent process, it can more effectively filter out irrelevant information and concentrate resources on identifying key features, thereby improving the detection accuracy of specific events.

[0042] Additional features will be described in part in the following description. For those skilled in the art, it will become apparent by reviewing the following and accompanying drawings, or it may be understood by the production or operation of the examples. The features of this specification may be implemented and obtained by practicing or using various aspects of the methods, tools, and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0044] Figure 1 It is a schematic diagram of an exemplary application scenario of a smart campus management system based on an artificial intelligence algorithm according to some embodiments of this specification;

[0045] Figure 2 is an exemplary module diagram of a smart campus management system based on an artificial intelligence algorithm according to some embodiments of this specification;

[0046] Figure 3 is an exemplary flow chart of a smart campus management method based on an artificial intelligence algorithm according to some embodiments of this specification;

[0047] Figure 4 It is a schematic diagram of the video content parsing process shown in some embodiments of this specification. DETAILED DESCRIPTION

[0048] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0049] It should be understood that the "system", "device", "unit" and / or "module" used in this specification is a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0050] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0051] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0052] The following is a detailed description of the smart campus management method and system based on artificial intelligence algorithm provided in the embodiments of this specification in conjunction with the accompanying drawings.

[0053] Figure 1 It is a schematic diagram of an exemplary application scenario of a smart campus management system based on artificial intelligence algorithms as shown in some embodiments of this specification.

[0054] Reference Figure 1 In some embodiments, the application scenario 100 of the smart campus management system based on artificial intelligence algorithm may include a monitoring video acquisition device 110, a storage device 120, a processing device 130, a terminal device 140 and a network 150. The various components in the application scenario 100 can be connected in a variety of ways. For example, the monitoring video acquisition device 110 can be connected to the storage device 120 and / or the processing device 130 through the network 150, or directly connected to the storage device 120 and / or the processing device 130. For another example, the storage device 120 can be directly connected to the processing device 130 or connected through the network 150. For another example, the terminal device 140 can be connected to the storage device 120 and / or the processing device 130 through the network 150, or directly connected to the storage device 120 and / or the processing device 130.

[0055] The monitoring video acquisition device 110 can be used to collect monitoring videos for any target area on campus. In the subsequent process, the current state of the target area can be identified based on the monitoring video. In some embodiments, in order to achieve all-round monitoring and shooting, the monitoring video acquisition device 110 can be set on the pan-tilt assembly. In some embodiments, the monitoring video acquisition device 110 can have an independent power supply, which can send the collected monitoring video to other components in the application scenario 100 (for example, storage device 120, processing device 130, terminal device 140) by wired or wireless (for example, Bluetooth, WiFi, etc.). In some embodiments, the application scenario 100 may include multiple (for example, two or more) monitoring video acquisition devices 110, and the multiple monitoring video acquisition devices 110 can collect monitoring videos for the same or different target areas.

[0056] In some embodiments, the surveillance video acquisition device 110 can send the surveillance video it collects to the storage device 120, the processing device 130, the terminal device 140, etc. through the network 150. In some embodiments, the surveillance video collected by the surveillance video acquisition device 110 can be processed by the processing device 130. For example, the processing device 130 can identify various events in the target area (such as fighting, theft, safety accidents, etc.) based on the surveillance video; for another example, the processing device 130 can provide data support for the smart campus management system based on the surveillance video (such as student attendance statistics, teacher teaching evaluation, campus traffic flow analysis, etc.). In some embodiments, the results obtained based on the surveillance video processing can be sent to the storage device 120 for recording, or sent to the terminal device 140 for feedback to the user (such as campus administrators or relevant staff).

[0057] The network 150 can facilitate the exchange of information and / or data. The network 150 may include any suitable network that can facilitate the exchange of information and / or data of the application scenario 100. In some embodiments, at least one component of the application scenario 100 (e.g., the monitoring video acquisition device 110, the storage device 120, the processing device 130, the terminal device 140) can exchange information and / or data with at least one other component in the application scenario 100 through the network 150. For example, the processing device 130 can obtain the monitoring video collected for any target area on campus from the monitoring video acquisition device 110 and / or the storage device 120 through the network 150. For another example, the processing device 130 can obtain user operation instructions from the terminal device 140 through the network 150. Exemplary operation instructions may include but are not limited to accessing the monitoring video, reading the results obtained based on the processing of the monitoring video, etc.

[0058] In some embodiments, the network 150 may be any form of wired or wireless network, or any combination thereof. As an example only, the network 150 may include a cable network, a wired network, an optical fiber network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC) network, etc., or any combination thereof. In some embodiments, the network 150 may include at least one network access point, and at least one component of the application scenario 100 may be connected to the network 150 through the access point to exchange data and / or information.

[0059] The storage device 120 may store data, instructions, and / or any other information. In some embodiments, the storage device 120 may store data obtained from the surveillance video acquisition device 110, the processing device 130, and / or the terminal device 140. For example, the storage device 120 may store the surveillance video acquired by the surveillance video acquisition device 110; for another example, the storage device 120 may store the result obtained by the processing device 130 based on the surveillance video. In some embodiments, the storage device 120 may store data and / or instructions used by the processing device 130 to execute or use to complete the exemplary method described in this specification. In some embodiments, the storage device 120 may include a large-capacity memory, a removable memory, a volatile read-write memory, a read-only memory (ROM), etc., or any combination thereof. Exemplary large-capacity memories may include magnetic disks, optical disks, solid-state disks, etc. In some embodiments, the storage device 120 may be implemented on a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc., or any combination thereof.

[0060] In some embodiments, the storage device 120 may be connected to the network 150 to communicate with at least one other component in the application scenario 100 (e.g., the surveillance video acquisition device 110, the processing device 130, the terminal device 140). At least one component in the application scenario 100 may access data, instructions, or other information stored in the storage device 120 through the network 150. In some embodiments, the storage device 120 may be directly connected or communicate with one or more components in the application scenario 100 (e.g., the surveillance video acquisition device 110, the terminal device 140). In some embodiments, the storage device 120 may be part of the surveillance video acquisition device 110 and / or the processing device 130.

[0061] The processing device 130 can process data and / or information obtained from the monitoring video acquisition device 110, the storage device 120, the terminal device 140 and / or other components of the application scenario 100. In some embodiments, the processing device 130 can obtain monitoring videos from any one or more of the monitoring video acquisition device 110, the storage device 120 or the terminal device 140, and process the monitoring videos to identify various events in the target area. In some embodiments, the processing device 130 can obtain pre-stored computer instructions from the storage device 120, and execute the computer instructions to implement the smart campus management method based on the artificial intelligence algorithm described in this specification.

[0062] In some embodiments, the processing device 130 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processing device 130 may be local or remote. For example, the processing device 130 may access information and / or data from the monitoring video acquisition device 110, the storage device 120, and / or the terminal device 140 via the network 150. For another example, the processing device 130 may be directly connected to the monitoring video acquisition device 110, the storage device 120, and / or the terminal device 140 to access information and / or data. In some embodiments, the processing device 130 may be implemented on a cloud platform. For example, a cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, a cloud-to-cloud cloud, a multi-cloud, etc., or any combination thereof.

[0063] The terminal device 140 can receive, send and / or display data. The received data may include data collected by the monitoring video acquisition device 110, data stored in the storage device 120, data processed by the processing device 130, etc. The sent data may include input data and instructions of users (such as campus administrators or relevant staff), etc. For example, the terminal device 140 can send the operation instructions input by the user to the monitoring video acquisition device 110 through the network 150 to control the monitoring video acquisition device 110 to perform corresponding data acquisition. For another example, the terminal device 140 can send the monitoring instructions input by the user to the processing device 130 through the network 150 to indicate the focus direction of the processing device 130 in the process of executing video content parsing.

[0064] In some embodiments, the terminal device 140 may include a mobile device 141, a tablet computer 142, a laptop computer 143, etc., or any combination thereof. For example, the mobile device 141 may include a mobile phone, a personal digital assistant (PDA), a dedicated mobile terminal, etc., or any combination thereof. In some embodiments, the terminal device 140 may include an input device (such as a keyboard, a touch screen), an output device (such as a display, a speaker), etc. In some embodiments, the processing device 130 may be a part of the terminal device 140.

[0065] It should be noted that the above description of the application scenario 100 is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the application scenario 100 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification. For example, the monitoring video acquisition device 110 may include more or fewer functional components.

[0066] Figure 2 is a schematic diagram of a module of a smart campus management system based on an artificial intelligence algorithm according to some embodiments of this specification. In some embodiments, Figure 2 The smart campus management system 200 based on artificial intelligence algorithm can be applied to Figure 1 The application scenario 100 shown, for example, can be configured in the form of software and / or hardware to the processing device 130 and / or the terminal device 140 to process the surveillance video collected by the surveillance video acquisition device 110 and identify various events in the target area based on the surveillance video.

[0067] Reference Figure 2 In some embodiments, the smart campus management system 200 based on artificial intelligence algorithm may include an acquisition module 210, a video frame extraction module 220, and a video content analysis module 230.

[0068] The acquisition module 210 may be used to acquire surveillance videos collected for a target area and monitoring instructions for monitoring a target event type.

[0069] The video frame extraction module 220 may be used to extract key frames from the surveillance video to obtain one or more target video frames.

[0070] The video content analysis module 230 can be used to input the one or more target video frames and the monitoring instructions into a trained video content analysis model to extract video content features and classify events, and obtain video content analysis text and event classification results corresponding to the one or more target video frames.

[0071] In an embodiment of the present application, the monitoring instruction is used to indicate the key focus areas of the video content parsing model for video content feature extraction and event classification. The video content parsing model includes a video frame feature extraction network, an event feature mapping network, a content parsing network, and an event classification network. The video content parsing model performs a first supervised training of video frame features and frame feature description text alignment on the video frame feature extraction network, event feature mapping network, and content parsing network of the initial video content parsing model through a graphic sample training set, and performs a second supervised training of video content feature extraction and event classification on the initial video content parsing model through a monitoring instruction sample training set.

[0072] Continue to refer to Figure 2 In an embodiment of the present application, the smart campus management system 200 based on artificial intelligence algorithm may also include a training module 240, which can be used to train the above-mentioned video content analysis model.

[0073] For more details about the above modules, please refer to other places in this manual (for example Figure 3~Figure 4 part and its related description) will not be repeated here.

[0074] It should be understood that Figure 2 The smart campus management system 200 based on artificial intelligence algorithm and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a dedicated design hardware. Those skilled in the art will understand that the above methods and systems can be implemented using computer executable instructions and / or included in processor control codes, such as carrier media such as disks, CDs or DVD-ROMs, programmable memories such as read-only memories (firmware), or data carriers such as optical or electronic signal carriers. Such codes are provided on. The system and its modules of this specification can not only be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above hardware circuits and software (e.g., firmware).

[0075] It should be noted that the above description of the smart campus management system 200 based on artificial intelligence algorithm is provided for illustrative purposes only and is not intended to limit the scope of this specification. It is understandable that for those skilled in the art, according to the description of this specification, without departing from this principle, the various modules can be arbitrarily combined, or a subsystem can be formed to connect with other modules. For example, Figure 2 The acquisition module 210, the video frame extraction module 220, the video content analysis module 230 and the training module 240 described in the above can be different modules in a system, or a module can realize the functions of two or more modules mentioned above. For another example, the smart campus management system 200 based on artificial intelligence algorithm can also include a preprocessing module ( Figure 2 (not shown in the figure), the preprocessing module can be used to preprocess the aforementioned surveillance video. Such variations are within the protection scope of this specification. In some embodiments, the aforementioned modules can be part of the processing device 130 and / or the terminal device 140.

[0076] Figure 3 is an exemplary flow chart of a smart campus management method based on an artificial intelligence algorithm according to some embodiments of this specification. In some embodiments, the smart campus management method based on an artificial intelligence algorithm can be executed by processing logic, which can include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (instructions running on a processing device to perform hardware simulation), etc. or any combination thereof. In some embodiments, Figure 3 One or more operations in the flowchart of the smart campus management method based on artificial intelligence algorithm shown in FIG. Figure 1 The processing device 130 and / or the terminal device 140 shown are implemented. For example, the smart campus management method based on the artificial intelligence algorithm can be stored in the storage device 120 in the form of a computer program or instruction, and called and / or executed by the processing device 130 and / or the terminal device 140.

[0077] Reference Figure 3 The smart campus management method based on artificial intelligence algorithm provided in the embodiment of the present application may include the following steps:

[0078] Step S310 , acquiring the surveillance video collected for the target area and the monitoring instruction for monitoring the target event type. In some embodiments, step S310 may be executed by the acquisition module 210 .

[0079] In the embodiment of the present application, the monitoring video acquisition device 110 can acquire monitoring videos for any target area in the campus, and the acquired monitoring videos can be stored in the storage device 120, and the acquisition module 210 can acquire the monitoring videos acquired for the target area from the storage device 120. In some embodiments of the present application, the acquisition module 210 can be connected to the monitoring video acquisition device 110 for communication, and the acquisition module 210 can directly acquire the monitoring videos acquired for the target area from the monitoring video acquisition device 110.

[0080] While acquiring the surveillance video, the acquisition module 210 can also acquire monitoring instructions for monitoring the target event type. The monitoring instructions corresponding to the target event type can be understood as a preset rule or pattern for guiding the identification and response of specific types of events in the subsequent process. For example, if the target event type is an abnormal behavior such as "fighting", the monitoring instruction may include a series of feature parameters, such as action behavior, number of people, duration, etc. These parameters will be used in the subsequent process to instruct the video content parsing module 230 to perform relevant feature extraction, so as to accurately identify the fighting event. For another example, if the target event type is an abnormal behavior such as "theft", the monitoring instruction may include moving speed (calculated by the time difference between different video frames and the moving distance of the target in different video frames), the characteristics of the carried items, behavior patterns, etc. These parameters will be used to identify theft in the surveillance video. For another example, when the target event type is "attendance statistics", the monitoring instruction may involve information such as the entry and exit of personnel in a specific time period, the number of people present, etc., so that the system can automatically record attendance and perform statistical analysis. When the target event type is "teaching evaluation", the monitoring instructions may include indicators such as classroom participation, student interaction, and teacher teaching quality. These indicators will be used to evaluate teaching effectiveness and student learning.

[0081] In an embodiment of the present application, the above-mentioned monitoring instructions may be pre-configured, and the corresponding monitoring instructions may be obtained by selecting a monitoring mode during actual use. In some embodiments of the present application, the monitoring instructions may exist in the form of text, which is used to indicate the focus of the video content analysis model in the subsequent process of performing video content feature extraction and event classification. Specifically, the monitoring instructions may provide the video content analysis model with task instructions for performing video content analysis and event classification under a preset monitoring mode or before and after association clues between different video frames, which may include but are not limited to feature summarization, video content description, or a mapping relationship between input and expected output, etc., thereby improving the accuracy and pertinence of the video content analysis model in the video analysis process through the monitoring instructions, thereby improving the reliability of the monitoring video analysis results. As an example only, in some embodiments of the present application, when the monitoring instruction corresponds to the recognition of "fighting" behavior, the monitoring instruction may be "analyzing whether there are multiple people (two or more) gathered in the video (or video frame) accompanied by aggressive actions, etc." When the monitoring instruction corresponds to "attendance statistics", the monitoring instruction may be "identify the number of people appearing in the classroom in the video (or video frame) and the corresponding identity of each person". In an embodiment of the present application, after identifying the identity of the person, the attendance of the students can be counted by comparing with the database.

[0082] It should be pointed out that in the embodiment of the present application, the monitoring instruction guides the video content parsing model to perform feature analysis based on this in the subsequent process, which can more effectively filter irrelevant information and concentrate resources to identify key features, thereby improving the detection accuracy of specific events. In the embodiment of the present application, the monitoring instruction can be customized according to different monitoring needs to adapt to various scenarios and environments, thereby ensuring the flexibility and efficiency of the smart campus management system. For example, in some embodiments, the monitoring instruction can be designed to identify specific campus activities, such as "detecting abnormal behavior of spectators in sports events" or "monitoring the learning status of students in the library". Through such customized monitoring instructions, the video content parsing model can more accurately identify task-related features, such as patterns of abnormal behavior, characteristics of learning status, etc., thereby achieving efficient detection and response to specific events. In addition, in some embodiments of the present application, the monitoring instruction can also be adjusted according to the real-time needs of campus management. For example, in an emergency, the monitoring instruction can quickly switch to the "search for missing students" mode to ensure campus safety.

[0083] Step S320 , extract key frames from the surveillance video to obtain one or more target video frames. In some embodiments, step S320 may be performed by the video frame extraction module 220 .

[0084] In the embodiment of the present application, considering that there may be still images in the surveillance video (i.e., the content of the two previous video frames is exactly the same), if all video frames are analyzed, it may lead to unnecessary waste of computing resources. Therefore, in the embodiment of the present application, the video frame extraction module 220 will first identify and exclude these still images, and only extract key frames from those video frames containing new information or changes.

[0085] Specifically, in the embodiment of the present application, based on the monitoring instruction, the video frame whose frame difference is greater than the corresponding preset frame difference threshold is used as the key frame. In other words, in the embodiment of the present application, the same or different frame difference thresholds can be set for different monitoring instructions to filter out the key frames in the monitoring video.

[0086] For example, when the mode corresponding to the monitoring instruction is to detect theft, the preset frame difference threshold can be set relatively low in order to capture subtle changes in movement; and when the monitoring instruction is to detect fighting, the preset frame difference threshold can be set relatively high to facilitate the identification of larger movements or scene changes. In this way, the system can flexibly adjust the extraction criteria of key frames according to different monitoring needs, thereby improving the accuracy and efficiency of surveillance video analysis. In addition, in some embodiments of the present application, the video frame extraction module 220 can also dynamically adjust the frame difference threshold according to factors such as light changes and camera angles in the actual monitoring environment to adapt to different monitoring scenarios.

[0087] In some embodiments of the present application, an average value of frame differences corresponding to the surveillance video may be calculated, and then video frames whose frame differences are greater than the average value may be used as the aforementioned key frames.

[0088] It can be understood that for the surveillance video collected for the target area, one or more key frames can be extracted in the above manner, and the one or more key frames can be analyzed as target video frames to obtain the video content analysis text and event classification results corresponding to each target video frame. Please refer to the following text for the specific process of analyzing the target video frame, which will not be described in detail here.

[0089] Step S330, input the one or more target video frames and the monitoring instructions into the trained video content analysis model to extract video content features and classify events, and obtain video content analysis text and event classification results corresponding to the one or more target video frames. In some embodiments, step S330 can be performed by the video content analysis module 230.

[0090] Specifically, refer to Figure 4In an embodiment of the present application, the video content parsing model may include a video frame feature extraction network, an event feature mapping network, a content parsing network, and an event classification network. In an embodiment of the present application, the video content parsing model can be obtained by performing a first supervised training of aligning video frame features and frame feature description texts on the video frame feature extraction network, the event feature mapping network, and the content parsing network of the initial video content parsing model through a graphic sample training set, and performing a second supervised training of video content feature extraction and event classification on the initial video content parsing model through a monitoring instruction sample training set. Among them, the initial video content parsing model can be understood as an untrained video content parsing model. The specific process of the first supervised training and the second supervised training can be referred to below, and will not be described in detail here.

[0091] Specifically, in an embodiment of the present application, the above-mentioned video frame feature extraction network can be used to extract features from the one or more target video frames to obtain video frame features corresponding to each target video frame; the event feature mapping network is used to convert the video frame features into frame feature description texts in a text embedding space; the content parsing network is constructed based on a pre-trained generative language model, and is used to obtain video content parsing texts and event content features corresponding to each target video frame based on the frame feature description texts and the monitoring instructions; the event classification network is used to perform classification and identification based on the event content features output by the content parsing network to obtain the event classification results.

[0092] In the embodiments of the present application, the aforementioned generative language model refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. The generative language model used in the embodiments of the present application may include but is not limited to pre-trained BERT models, GPT models, T5 models, etc. These models can capture rich language regularities and semantic information by pre-training on a large-scale corpus, thereby demonstrating powerful language understanding and generation capabilities in specific tasks. For example, the BERT model can represent the bidirectional contextual relationship of learning language from large-scale text through a bidirectional encoder, while the GPT model can utilize deep generative pre-training technology and be trained in an autoregressive manner to generate coherent and contextual text. The T5 model unifies various NLP tasks into text-to-text conversion problems, and can adapt to a variety of different downstream tasks through large-scale pre-training and fine-tuning. These models can effectively assist the system in parsing and classifying surveillance video content and events in the embodiments of the present application, thereby realizing the intelligence and automation of the smart campus management system.

[0093] Specifically, in some embodiments of the present application, a video frame feature extraction network can be used to extract features from a target video frame to obtain slice-level video frame features. The slice-level video frame features can be understood as video frame features corresponding to the same video frame that contain features corresponding to multiple slice regions, wherein each slice region can correspond to a specific part of the video frame. In this way, the video frame feature extraction network can capture detailed information in the video content more meticulously, thereby improving the accuracy of event classification and parsing. For example, in a surveillance video, if it is necessary to identify and track specific people or objects, slice-level feature extraction can provide richer spatial information to help the system locate and identify targets more accurately. In addition, this feature extraction method can also be used for video content retrieval. By comparing slice-level features of different video frames, the system can quickly find similar or related content segments. Figure 4 As shown, in an embodiment of the present application, the video frame feature extraction network can perform feature extraction on the target video frame based on the aforementioned monitoring instructions, thereby making the feature extraction process more targeted.

[0094] The event feature mapping network can be used to perform feature mapping with the aforementioned video frame features as input, align the video frame features and text features, and obtain a frame feature description text that can be understood by the content parsing network. In other words, in an embodiment of the present application, the event feature mapping network can convert the video frame features into a frame feature description text in a text embedding space.

[0095] Continue to refer to Figure 4In the embodiment of the present application, the content parsing network can be used to take the frame feature description text obtained by the aforementioned monitoring instruction and event feature mapping network as input, and use the monitoring instruction as the guiding information to perform content analysis on the frame feature description text, thereby obtaining event content features that can express the video frame content analysis indicated by the monitoring instruction, and mapping the event content features through its output layer to output the video content parsing text. It should be noted that in the embodiment of the present application, the video content parsing text can be understood as the text data generated for the monitoring instruction to describe the content of the monitoring video frame. Exemplarily, the video content analysis text for the aforementioned “analyzing whether there are multiple people (two or more) gathering with aggressive actions in the video (or video frame)” can be “the surveillance video shows that three people gathered on the basketball court in the south area of ​​the campus, and one of them attacked another person with his fist”; for another example, the video content analysis text for the aforementioned “identifying the number of people appearing in the classroom in the video (or video frame) and the corresponding identity of each person” can be “the surveillance video shows that there are 20 students and 1 teacher in the classroom, among which the teacher is Teacher xx, and the students include 15 boys and 5 girls, A~T respectively (A~T can be the student's name or corresponding number, such as student number)”. In an embodiment of the present application, the content analysis network can perform content analysis on the frame feature description text with the aforementioned monitoring instruction as the guiding information, so as to obtain the event content features that can express the video frame content analysis indicated by the monitoring instruction. In some embodiments of the present application, the analysis process may include analyzing whether each piece of information in the frame feature description text is related to the event type corresponding to the monitoring instruction, and if not, omitting it. For example, for the aforementioned video content analysis text "The surveillance video shows that there are 20 students and 1 teacher in the classroom. The teacher is Teacher xx, and the students include 15 boys and 5 girls, A~T respectively", the content analysis network can use the aforementioned monitoring instructions as guidance information to perform content analysis on the frame feature description text, and obtain the event content feature "There are 20 students and 1 teacher in total. The teacher is Teacher xx, and the students are A~T respectively".

[0096] Further, the event classification network can be used to take the event content features as input and output the event classification results, and the event classification results are used to characterize the preset category to which the target video frame belongs under the preset monitoring mode. For example, in the "fight monitoring" mode, the event content features detected in the video frame can be matched with the preset categories in the "fight monitoring" mode to determine whether the current video frame meets the characteristics of the "fight" event. If it meets, the event classification network will output the corresponding classification results, such as "fighting event occurred", and trigger the corresponding alarm mechanism to notify the management personnel to intervene. For another example, in the "theft monitoring" mode, the event content features detected in the video frame can be matched with the preset categories in the "theft monitoring" mode to determine whether theft has occurred. If the video content meets the characteristics of the theft event, the event classification network will output the classification result of "theft event occurred" and activate the alarm system to promptly notify the security personnel to take measures. For another example, in the "attendance statistics" mode, the number of people and personnel identity reflected in the event content features can be compared with the preset number of people and personnel identity information to obtain the attendance of each student. It can be understood that through the above method, the smart campus management system can realize real-time monitoring and classification of various events on campus, improving campus safety and management efficiency.

[0097] It should be noted that the above event monitoring is only an example. In the actual use of the smart campus management system, other events can be identified and monitored in a similar manner.

[0098] Specifically, in an embodiment of the present application, the video frame features obtained by extracting features from the aforementioned one or more target video frames through a video frame feature extraction network may include surface features and depth features of the target video frames. In some embodiments, the video frame feature extraction network may be obtained by training a pre-configured indication information feature extraction unit and a video frame feature extraction unit based on a self-attention mechanism through a training set of image and text samples.

[0099] Specifically, the text data (such as attention direction information, label text, etc.) in the image and text sample training set can be input into the indication information feature extraction unit for feature extraction to obtain text features, and the image data (such as video frames) in the image and text sample training set can be input into the video frame feature extraction unit for feature extraction to obtain image features. Further, the similarity between the text features and the image features can be calculated, and the text features and the image features can be matched in a classification recognition-based manner, and the matching results of the two can be output, and then the network loss value is determined based on the matching results to adjust the network parameters through back propagation until the pre-training conditions are met. The video frame feature extraction unit that meets the pre-training conditions is determined as a video frame feature extraction network. In an embodiment of the present application, the above-mentioned image and text sample training set can be composed of a number of sample monitoring video frames and their corresponding attention direction information and description information (i.e., label information).

[0100] In some embodiments of the present application, the video frame feature extraction unit can be constructed based on a convolutional neural network (CNN) and a self-attention mechanism. For example, a visual module such as a CLIP model (Contrastive Language-Image Pre-Training) or CLIP ViT-L / 14 can be adopted.

[0101] After the target video frame is feature extracted through the above-mentioned video frame feature extraction network to obtain the video frame features, the video frame features can be input into the event feature mapping network for feature mapping to map the video frame features to the text embedding space of the content parsing network to obtain the frame feature description text. Specifically, in an embodiment of the present application, after being processed by the event feature mapping network, the aforementioned slice-level features can be mapped to text token features to improve the accuracy of feature expression, so as to facilitate the content parsing network to perform frame content parsing in the subsequent process.

[0102] In some embodiments of the present application, the event feature mapping network may include a first mapping layer and a second mapping layer (i.e., constructed based on a 2-layer Transformer). The event feature mapping network may be specifically used to: input the video frame features into the first mapping layer for slice-level feature mapping, so as to map the video frame features to the word embedding space in units of slices to obtain slice embedding features; input the slice embedding features into the second mapping layer for mining and extraction of contextual information to obtain frame feature description text.

[0103] Specifically, in an embodiment of the present application, the first mapping layer can be used to align video frame features and text word feature representations to map slice-level video frame features to token-level slice embedding features. In an embodiment of the present application, the first mapping layer can be equivalent to a character-level Transformer, and the slice embedding feature can be equivalent to the sentence representation obtained after encoding and extracting the text sentence. The second mapping layer can be used to mine and extract the contextual information of each token corresponding to the aforementioned slice embedding feature based on the self-attention mechanism. In an embodiment of the present application, the second mapping layer can be equivalent to a sentence-level Transformer, and the frame feature description text is equivalent to the text representation after the sentence representation is enhanced with contextual information. It can be understood that in an embodiment of the present application, the first mapping layer is used to achieve preliminary alignment of video frame features and text features, and then the slice embedding features containing scattered slice feature expressions are converted into frame feature description text that can express the overall content of the video frame through the second mapping layer, which can improve the expression accuracy of the frame feature description text to a certain extent.

[0104] In some embodiments, the aforementioned event feature mapping network may include a first mapping layer and a second mapping layer, and both layers of transformers are used to mine and extract the contextual information of the input features based on the self-attention mechanism, that is, the video frame features are input into the first mapping layer and the output obtained after cross-extraction is used as the input of the second mapping layer, and the frame feature description text is obtained after another cross-extraction. It can be understood that in this way, the mapping of the video frame feature space and the text word embedding space is achieved through feature extraction strengthened by two layers of contextual information, which can improve the feature alignment of the content parsing network and achieve accurate frame content parsing.

[0105] Furthermore, the frame feature description text obtained in the above process and the monitoring instructions corresponding to the target event type can be input into the content analysis network for frame content analysis to obtain the corresponding event content features and video content analysis text.

[0106] Specifically, in an embodiment of the present application, the video content analysis text can be obtained by mapping the event content features to feature text based on the output layer of the content analysis network. The monitoring instruction can first be processed by text feature representation to generate the corresponding indication embedding feature. In other words, the monitoring instruction can be mapped to the text word embedding space of the content analysis network and aligned with the frame feature description text. In an embodiment of the present application, the text feature representation processing can be implemented by a text feature representation layer separately arranged before the content analysis network, or it can also be implemented by a text feature representation layer in the content analysis network. The spliced ​​data obtained by splicing the frame feature description text with the aforementioned indication embedding feature can generate event content features after feature extraction by the content analysis network, and obtain the video content analysis text after mapping by the output layer. It should be noted that in an embodiment of the present application, the event content feature can fuse the frame content of the target video frame and the content indicated by the monitoring instruction, so as to realize the tendency analysis and expression of the video content in the direction of attention indicated by the monitoring instruction, thereby better satisfying the video frame content analysis and classification under the preset monitoring mode.

[0107] In an embodiment of the present application, the aforementioned event content features can be input into an event classification network for classification and identification to obtain an event classification result. Specifically, the event classification network can be connected before the output layer of the video content analysis model, which can take the event content features as input and map them to corresponding preset categories, thereby realizing the simultaneous output of the video content analysis text and event classification results of the video content analysis model for the target video frame.

[0108] Based on some or all of the above implementation modes, in an embodiment of the present application, the video content analysis model can be obtained by performing a first supervised training of aligning video frame features and frame feature description texts on the video frame feature extraction network, event feature mapping network, and content analysis network of the initial video content analysis model through a graphic sample training set, and performing a second supervised training of video content feature extraction and event classification on the initial video content analysis model through a monitoring instruction sample training set. Through the above training method, segmented training of the video frame feature extraction network, event feature mapping network, and event classification network can be achieved, thereby improving the model performance and the convergence efficiency of the training process to a certain extent. The training process of the video content analysis model involved in the embodiment of the present application is described below.

[0109] In an embodiment of the present application, a first image-text sample training set corresponding to a routine event, a second image-text sample training set corresponding to an event of interest, and an indication information training set can be obtained. Among them, routine events refer to common and normal events in a campus environment, such as classroom interaction between teachers and students, student recess activities, and normal walking of students on campus. Events of interest refer to those events that require special attention and are of great significance to campus management, such as the teaching management events such as attendance statistics and teaching evaluation mentioned above, as well as campus safety events such as fighting and theft. The indication information training set contains various indication information related to the first image-text sample training set and / or the second image-text sample training set during the training process, such as sample monitoring instructions related to events of interest and corresponding output result annotations. In an embodiment of the present application, the first image-text sample training set can be formed by obtaining surveillance video frames corresponding to several routine events and their corresponding description information. Similarly, the second image-text sample training set can be formed by obtaining surveillance video frames corresponding to several events of interest and their corresponding description information.

[0110] Specifically, in an embodiment of the present application, the first graphic sample training set corresponding to a regular event may include a first sample video frame, first focus direction information, and a first content label text corresponding to the first sample video frame, where the first content label text is the expected parsing result corresponding to the video frame content description based on the first focus direction information. For example, the first focus direction information may be information related to the video content indicated for the first sample video frame, and the first content label text may be the output result for the first focus direction information.

[0111] In an embodiment of the present application, the second image and text sample training set corresponding to the event of interest may include a second sample video frame, second attention direction information, a second content label text corresponding to the second sample video frame, and an event classification label corresponding to the second sample video frame. The second content label text is the expected parsing result corresponding to the video frame content description based on the second attention direction information.

[0112] Similar to the monitoring instruction, in the embodiment of the present application, both the first focus direction information and the second focus direction information can be used to indicate the focus direction of the content parsing network when performing frame content parsing. The first focus direction information can be used to instruct the video content parsing model to perform frame content parsing on the first sample video frame and output a sample parsing result that is close to the first content label text; the second focus direction information is used to instruct the video content parsing model to perform frame content parsing on the second sample video frame and output a sample parsing result that is close to the second content label text.

[0113] In an embodiment of the present application, the first focus direction information corresponding to the same first sample video frame and / or the second focus direction information corresponding to the same second sample video frame may include a monitoring instruction for at least one direction, wherein the monitoring instruction for each direction includes indication information of multiple dimensions, and the indication information of the multiple dimensions may include an intra-frame target recognition indication, a frame picture content attribute extraction indication, and a frame picture content derivation indication, etc.

[0114] Among them, the in-frame target recognition indication can be used to instruct the content parsing network to perform in-frame target recognition, and the first content label text or the second content label text corresponding to the in-frame target recognition indication can have a unique true value; wherein, the in-frame target refers to the object element appearing in the video frame, such as the person, animal, plant, object or scenery appearing in the image, and the in-frame target recognition indication and the corresponding content label text can form a dialogue form, and the in-frame target recognition indication can be used to raise questions related to the target object element in the video frame, which can be a question with a clear output result, and the content label text can provide a unique true value output result for the answer to the corresponding question. Exemplarily, the in-frame target recognition indication can include but is not limited to questions related to object type, object position, object action, object quantity, relative position between objects, etc.

[0115] The frame picture content attribute extraction indication is used to guide the content parsing network to extract and describe the frame picture content attributes. The first content label text or the second content label text corresponding to the frame picture content attribute extraction indication has the attribute of an open output result, which can be used to comprehensively describe the target video frame to include elements, scene attributes, activity attributes, time attributes, etc. in the video frame.

[0116] The frame content derivation indication can be used to guide the content parsing network to derive and describe the frame content, and perform further reasoning analysis based on the video content itself. In an embodiment of the present application, the frame content derivation indication can be related to the corresponding event of interest, and is used to raise questions that match the event of interest. For example, in a fight monitoring mode, the frame content derivation indication can be "analyze whether there is a possibility of a fight in the video frame", and the corresponding content label text can be "impossible, because there is only one person in the current video frame", or "possible, because, because it is detected in the video frame that two people seem to be arguing".

[0117] It can be understood that in an embodiment of the present application, by setting the above-mentioned types of first focus direction information and second focus direction information, the video content parsing model can be promoted to understand the video content from different directions and / or different dimensions, thereby realizing the description of the surface features and deep meaning of the target video frame, improving the accuracy of frame content parsing, and thereby improving the accuracy of the video frame content description text and subsequent event classification results, as well as improving the matching with the current monitoring mode.

[0118] In an embodiment of the present application, the first supervised training of aligning video frame features and frame feature description texts can be performed on the video frame feature extraction network, event feature mapping network, and content analysis network of the initial video content analysis model based on the first image and text sample training set corresponding to regular events and the second image and text sample training set corresponding to events of interest. Specifically, during the first supervised training process, the network parameters of the content analysis network can be fixed and the network parameters of the video frame feature extraction network and the event feature mapping network can be adjusted until the first preset training condition is met.

[0119] Furthermore, the initial video content parsing model that meets the first preset training condition can be subjected to second supervised training for video content feature extraction and event classification based on the second graphic sample training set corresponding to the event of interest and the monitoring instruction sample training set formed by the instruction information training set. During the second supervised training process, the network parameters of the video frame feature extraction network and the event feature mapping network are fixed and the network parameters of the content parsing network and the event classification network are adjusted until the second preset training condition is met. Finally, the initial video content parsing model that meets the second preset training condition is used as the trained video content parsing model.

[0120] Specifically, in some embodiments, the first supervised training process may include the following steps:

[0121] S11, using the first sample video frame and / or the second sample video frame as input of a video frame feature extraction network to perform video frame feature extraction, and obtaining first sample video frame features corresponding to the first sample video frame and / or second sample video frame features corresponding to the second sample video frame.

[0122] S12, using the first sample video frame features and / or the second sample video frame features as input of the event feature mapping network to perform event feature mapping, thereby mapping the first sample video frame features and / or the second sample video frame features to the text embedding space corresponding to the content parsing network, and obtaining the first frame feature description text corresponding to the first sample video frame and / or the second frame feature description text corresponding to the second sample video frame.

[0123] S13, using the data pair consisting of the first frame feature description text and the first focus direction information and / or the data pair consisting of the second frame feature description text and the second focus direction information as the input of the content parsing network to perform frame content parsing, and obtain the first sample parsing result corresponding to the first sample video frame and / or the second sample parsing result corresponding to the second sample video frame.

[0124] S14, determining a first network loss value according to a difference between the first sample parsing result and the first content label text and a difference between the second sample parsing result and the second content label text.

[0125] S15, fixing the network parameters of the content parsing network, and optimizing the network parameters of the video frame feature extraction network and the event feature mapping network according to the first network loss value, until the first preset training condition is met and the first supervised training is completed.

[0126] Specifically, in the first supervised training stage, the first image and text sample training set corresponding to regular events and / or the second image and text sample training set corresponding to attention events can be used as training data. In a single iteration, multiple first sample video frames and / or second sample video frames can be input into the video frame feature extraction network to obtain corresponding video frame features respectively, and generate first frame feature description text corresponding to the first sample video frame and / or second frame feature description text corresponding to the second sample video frame through feature mapping.

[0127] Furthermore, after the first indication embedding feature corresponding to the first focus direction information is concatenated with the first frame feature description text, feature extraction based on frame content analysis is performed through a content analysis network, so as to generate a first sample analysis result. Similarly, after the second indication embedding feature corresponding to the second focus direction information is concatenated with the second frame feature description text, feature extraction based on frame content analysis is performed through a content analysis network, so as to generate a second sample analysis result.

[0128] The first network loss value can be determined by fusing the first loss and the second loss, wherein the first loss can be obtained based on the difference between the first sample parsing result and the first content label text, and the second loss can be obtained based on the difference between the second sample parsing result and the second content label text. In an embodiment of the present application, the network parameters of the video frame feature extraction network and the event feature mapping network can be adjusted by the first network loss value, so that they can continuously learn the relationship and knowledge between the video frames corresponding to regular events and the video frames corresponding to the events of interest and the frame feature description text, thereby improving the accuracy and reliability of the alignment between computer vision and language description text.

[0129] Further, in an embodiment of the present application, the initial video content analysis model that meets the first preset training condition can be subjected to second supervised training for video content feature extraction and event classification based on the second graphic sample training set corresponding to the event of interest and the monitoring instruction sample training set formed by the instruction information training set. During the second supervised training process, the network parameters of the video frame feature extraction network and the event feature mapping network can be fixed and the network parameters of the content analysis network and the event classification network can be adjusted until the second preset training condition is met.

[0130] In some embodiments of the present application, the second supervised training process may include the following steps:

[0131] S21, taking the second sample video frame as an input of a video frame feature extraction network that meets a first preset training condition to perform video frame feature extraction to obtain a third sample video frame feature.

[0132] S22, using the third sample video frame feature as an input of an event feature mapping network that meets the first preset training condition to perform event feature mapping to obtain a third frame feature description text.

[0133] S23, using the data pair consisting of the second frame feature description text and the second focus direction information, and the sample monitoring instruction as input of the content parsing network to perform frame content parsing, and obtain the third event content feature, the third sample parsing result and the output result text.

[0134] S24, inputting the third event content feature into the event classification network for classification and identification to obtain a sample event classification result.

[0135] S25, determining a second network loss value according to the third sample parsing result, the second content label text, the output result text, the output result annotation, the sample event classification result and the event classification label.

[0136] S26, fixing the network parameters of the video frame feature extraction network and the event feature mapping network, and training the content parsing network and the event classification network according to the second network loss value until the second preset training condition is met to complete the second supervised training.

[0137] Specifically, in the second supervised training stage, the second image and text sample training set and the instruction information training set corresponding to the event of interest can be used as training data to learn knowledge related to the event of interest, thereby improving the matching degree between the video content parsing model and the target event monitoring mode.

[0138] In an embodiment of the present application, for the second image and text sample training set corresponding to the event of interest, similar to the first supervised training stage, the second sample video frame is input into the video frame feature extraction network, and after feature extraction, the third sample video frame feature can be obtained, and then after feature mapping, the third frame feature description text can be obtained; further, the third frame feature description text can be spliced ​​with the third indication embedding feature corresponding to the second focus direction information, so as to perform subsequent feature extraction based on frame content analysis, and generate the third event content feature and the third sample analysis result. Among them, the third event content feature can be understood as the sample event classification result obtained after passing through the event classification network, which is similar to the aforementioned event content feature and will not be repeated here.

[0139] In an embodiment of the present application, for the instruction information training set, the sample monitoring instructions can be input into the content parsing network for feature embedding and feature extraction to generate an output result text, thereby training the language knowledge comprehension ability of the content parsing network.

[0140] In an embodiment of the present application, the aforementioned second network loss value can be obtained by fusing the third loss, the fourth loss and the fifth loss, wherein the third loss can be determined based on the difference between the third sample parsing result and the second content label text, the fourth loss can be determined based on the difference between the output result text and the output result annotation, and the fifth loss can be determined based on the difference between the sample event classification result and the event classification label.

[0141] It should be noted that in the embodiment of the present application, during the training process of the first supervised training stage and the second supervised training stage, the stochastic gradient descent method (SGD for short) can be used to solve the network model parameters (such as weight parameter w and bias parameter b, etc.). In each iteration, the network loss value can be calculated and back-propagated to the network model to calculate the gradient and optimize the network parameters. More details about model training can be regarded as prior art and will not be discussed in detail in this specification.

[0142] In an embodiment of the present application, an initial video content parsing model that meets the second preset training condition can be used as a trained video content parsing model to parse one or more target video frames extracted from the surveillance video collected for the target area, and obtain a video content parsing text and event classification result corresponding to each target video frame. In some embodiments, the video content parsing text can be stored as judgment basis information. It should be pointed out that in an embodiment of the present application, through the above-mentioned video content parsing model, different target events can be identified under the guidance of monitoring instructions, avoiding the need for separate modeling and training of different event types, thereby reducing the complexity and development and maintenance costs of the system to a certain extent, and improving the intelligence of the system.

[0143] It can be understood that in the embodiment of the present application, the event classification result corresponding to each target video frame can reflect whether there is a target event targeted by the corresponding monitoring instruction in the video frame. In the embodiment of the present application, when the monitoring instruction contains indication information of multiple directions, the event classification result can include classification result data of multiple directions.

[0144] In some embodiments of the present application, when monitoring abnormal behaviors such as fighting and theft, it is possible to determine whether there is an abnormal event in the surveillance video by using the event classification results corresponding to a plurality of (e.g., two or more) consecutive target video frames. For example, in some embodiments of the present application, when at least 2 / 3 of the data in the plurality of event classification results corresponding to the plurality of target video frames reflect the presence of an abnormal event, it is determined that an abnormal event exists in the target area.

[0145] In summary, the beneficial effects that may be brought about by the embodiments of this specification include but are not limited to: (1) In the smart campus management system based on artificial intelligence algorithm provided in some embodiments of this specification, by performing content analysis and event classification on the surveillance video collected for the target area, the intelligence and automation level of the smart campus management system can be realized, and the timeliness and comprehensiveness of campus safety management can be improved; (2) In the smart campus management system based on artificial intelligence algorithm provided in some embodiments of this specification, a video content analysis model is constructed by a video frame feature extraction network, an event feature mapping network, a content analysis network and an event classification network, and by performing a first supervised training and a second supervised training on it, the trained video content analysis model can be used to identify different target events under the guidance of monitoring instructions, avoiding the need to model and train different event types separately, thereby reducing the complexity and development and maintenance costs of the system to a certain extent and improving the intelligence of the system; (3) In the smart campus management system based on artificial intelligence algorithm provided in some embodiments of this specification, by guiding the video content analysis model to perform feature analysis based on this in the subsequent process through monitoring instructions, irrelevant information can be filtered more effectively, and resources can be concentrated on identifying key features, thereby improving the detection accuracy of specific events.

[0146] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or may be any other beneficial effects that may be obtained.

[0147] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0148] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0149] In addition, it will be understood by those skilled in the art that various aspects of this specification may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of this specification may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, various aspects of this specification may be represented as a computer product located in one or more computer-readable media, which includes computer-readable program code.

[0150] A computer storage medium may include a propagated data signal containing computer program code, for example, in baseband or as part of a carrier wave. The propagated signal may be in a variety of forms, including electromagnetic, optical, etc., or a suitable combination. A computer storage medium may be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit the program for use. The program code on the computer storage medium may be transmitted via any suitable medium, including radio, cable, fiber optic cable, RF, or similar media, or any combination of the above media.

[0151] The computer program codes required for the operation of each part of this specification can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C language, Visual Basic, Fortran2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages, etc. The program code can be run entirely on the user's computer, or run on the user's computer as an independent software package, or run partly on the user's computer and partly on a remote computer, or run entirely on a remote computer or processing device. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0152] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing processing device or mobile device.

[0153] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0154] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining the digits. Although the numerical domains and parameters used to confirm the breadth of the range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0155] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification is hereby incorporated by reference in its entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0156] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A smart campus management system based on artificial intelligence algorithm, characterized in that: include: An acquisition module is used to acquire the surveillance video collected for the target area and the monitoring instructions for monitoring the target event type; A video frame extraction module is used to extract key frames from the surveillance video to obtain one or more target video frames; The video content analysis module is used to input the one or more target video frames and the monitoring instructions into the trained video content analysis model to extract video content features and classify events, and obtain the video content analysis text and event classification results corresponding to the one or more target video frames; wherein, The monitoring instruction is used to indicate the key focus areas of the video content parsing model for video content feature extraction and event classification. The video content parsing model includes a video frame feature extraction network, an event feature mapping network, a content parsing network and an event classification network. The video content parsing model is obtained by performing a first supervised training of aligning video frame features and frame feature description text on the video frame feature extraction network, the event feature mapping network and the content parsing network of the initial video content parsing model through a graphic sample training set, and performing a second supervised training of video content feature extraction and event classification on the initial video content parsing model through a monitoring instruction sample training set.

2. The smart campus management system based on artificial intelligence algorithm as claimed in claim 1, characterized in that: The video frame extraction module is specifically used to: based on the monitoring instruction, take the video frame whose frame difference is greater than the corresponding preset frame difference threshold as a key frame.

3. The smart campus management system based on artificial intelligence algorithm as claimed in claim 1, characterized in that: The video frame feature extraction network is used to extract features from the one or more target video frames to obtain video frame features corresponding to each target video frame; the event feature mapping network is used to convert the video frame features into frame feature description text in a text embedding space; the content parsing network is constructed based on a pre-trained generative language model and is used to obtain video content parsing text and event content features corresponding to each target video frame based on the frame feature description text and the monitoring instructions; The event classification network is used to perform classification and identification based on the event content features to obtain the event classification result.

4. The smart campus management system based on artificial intelligence algorithm as claimed in claim 3 is characterized in that: The event feature mapping network includes a first mapping layer and a second mapping layer, and the event feature mapping network is specifically used for: Inputting the video frame features into the first mapping layer for slice-level feature mapping, so as to map the video frame features to a word embedding space in units of slices to obtain slice embedding features; The slice embedding feature is input into the second mapping layer to mine and extract the previous and next associated information, so as to obtain the frame feature description text.

5. The smart campus management system based on artificial intelligence algorithm as claimed in claim 3, characterized in that: The video frame feature extraction network is obtained by performing image-text matching training on a pre-configured indication information feature extraction unit and a video frame feature extraction unit based on a self-attention mechanism through an image-text sample training set.

6. The smart campus management system based on artificial intelligence algorithm as claimed in any one of claims 1 to 5, characterized in that: The system further comprises a training module, wherein the training module is specifically used for: Obtaining a first image and text sample training set corresponding to a conventional event, a second image and text sample training set corresponding to a concerned event, and an instruction information training set consisting of a plurality of monitoring instructions; Based on the first image-text sample training set corresponding to the regular event and the second image-text sample training set corresponding to the event of interest, the video frame feature extraction network, the event feature mapping network and the content analysis network of the initial video content analysis model are subjected to first supervised training for aligning video frame features and frame feature description texts, and in the first supervised training process, the network parameters of the content analysis network are fixed and the network parameters of the video frame feature extraction network and the event feature mapping network are adjusted until a first preset training condition is met; Based on the second image and text sample training set corresponding to the event of interest and the monitoring instruction sample training set formed by the indication information training set, a second supervised training of video content feature extraction and event classification is performed on the initial video content parsing model that meets the first preset training condition, and in the second supervised training process, the network parameters of the video frame feature extraction network and the event feature mapping network are fixed and the network parameters of the content parsing network and the event classification network are adjusted until the second preset training condition is met; The initial video content parsing model that meets the second preset training condition is used as the trained video content parsing model.

7. The smart campus management system based on artificial intelligence algorithm as claimed in claim 6, characterized in that: The first graphic sample training set corresponding to the regular event includes a first sample video frame, first attention direction information, and a first content label text corresponding to the first sample video frame. The second graphic sample training set corresponding to the attention event includes a second sample video frame, second attention direction information, and a second content label text corresponding to the second sample video frame. The first attention direction information and the second attention direction information are both used to guide the content parsing network to focus on the direction when parsing the frame content. The first content label text is an expected parsing result corresponding to the video frame content description based on the first attention direction information, and the second content label text is an expected parsing result corresponding to the video frame content description based on the second attention direction information. The first supervised training process comprises the following steps: Using the first sample video frame and / or the second sample video frame as input of the video frame feature extraction network to perform video frame feature extraction, to obtain first sample video frame features corresponding to the first sample video frame and / or second sample video frame features corresponding to the second sample video frame; Using the first sample video frame feature and / or the second sample video frame feature as input of the event feature mapping network to perform event feature mapping, thereby mapping the first sample video frame feature and / or the second sample video frame feature to a text embedding space corresponding to the content parsing network, and obtaining a first frame feature description text corresponding to the first sample video frame and / or a second frame feature description text corresponding to the second sample video frame; Using the data pair consisting of the first frame feature description text and the first focus direction information and / or the data pair consisting of the second frame feature description text and the second focus direction information as inputs of the content parsing network to perform frame content parsing, and obtaining a first sample parsing result corresponding to the first sample video frame and / or a second sample parsing result corresponding to the second sample video frame; Determine a first network loss value according to a difference between the first sample parsing result and the first content label text and a difference between the second sample parsing result and the second content label text; The network parameters of the content parsing network are fixed, and the network parameters of the video frame feature extraction network and the event feature mapping network are optimized according to the first network loss value, until the first preset training condition is met and the first supervised training is completed.

8. The smart campus management system based on artificial intelligence algorithm as claimed in claim 7, characterized in that: The second graphic sample training set corresponding to the event of interest also includes an event classification label corresponding to the second sample video frame, the instruction information training set includes sample monitoring instructions and output result labels corresponding to the sample monitoring instructions, and the second supervised training process includes the following steps: Using the second sample video frame as an input of a video frame feature extraction network that meets the first preset training condition to extract video frame features, thereby obtaining third sample video frame features; Using the third sample video frame feature as an input of an event feature mapping network that meets the first preset training condition to perform event feature mapping, and obtaining a third frame feature description text; Using the data pair consisting of the second frame feature description text and the second focus direction information, and the sample monitoring instruction as inputs of the content parsing network to perform frame content parsing, and obtain a third event content feature, a third sample parsing result, and an output result text; Inputting the third event content feature into the event classification network for classification and identification to obtain a sample event classification result; Determine a second network loss value according to the third sample parsing result, the second content label text, the output result text, the output result annotation, the sample event classification result, and the event classification label; The network parameters of the video frame feature extraction network and the event feature mapping network are fixed, and the network parameters of the content parsing network and the event classification network are optimized according to the second network loss value, until the second preset training condition is met, and the second supervised training is completed.

9. The smart campus management system based on artificial intelligence algorithm as claimed in claim 7, characterized in that: The first focus direction information and the second focus direction information both include monitoring instructions for at least one direction, wherein the monitoring instructions for each direction include indication information of multiple dimensions, and the indication information of multiple dimensions include intra-frame target recognition indication, frame picture content attribute extraction indication and frame picture content derivation indication.

10. The smart campus management system based on artificial intelligence algorithm as claimed in claim 9, characterized in that: The intra-frame object recognition indication is used to guide the content parsing network to perform intra-frame object recognition; The frame picture content attribute extraction instruction is used to guide the content parsing network to extract and describe the frame picture content attribute, and the frame picture content attribute includes at least one of a time attribute, a scene attribute, and an activity attribute; The frame picture content derivation indication is used to guide the content parsing network to perform frame picture content derivation and description.

Images